inscatolati - Get out of the box! :)

• ISOLINUX is able to boot from the CD directly, without anything to worry about or anything strange to do.

• ISOLINUX, however, is quite limited in respect with what it can do once the CD is booted.

• GRUB is more flexible, and looks nice. Older versions of GRUB had no direct support to boot ISO images. http://www.lrz-muenchen.de/~bernhard/grub-chain-cd.html contain useful info. For 0.93, there was a patch available, which can easily be found on google (by searching something like El-Torito grub support).

  Recent versions of GRUB, however, seems to have a very good support (since march 2004, accordingly to the Changelog). There is plenty of documentation in the info manual (info grub).

• both GRUB and ISOLINUX do support Graphic Splash Screens.

If you run the "mount" command alone, you should get the list of the mounted partitions, something like:

/dev/sda1 on / type ext3 (rw,errors=remount-ro) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw)

Sometimes, especially when the root partition contains some errors, or has been mounted/remounted read only, the output of mount is screwd up: read-only partitions are shown as being read-write, unmounted partitions are still shown as mounted, and so on.

The problem is due to the fact that mount uses the file /etc/mtab for book keeping, writing there whenever a partition is mounted, unmounted, or remounted with different options.

If /etc/mtab is read only, for example because the root partition is read only, mount and many system tools may get confused about the status of the partitions, and may not be able to update their status. Symptoms are: partitions being unmounted without errors and still being shown by the mount command, read-only partitions shown as read-write, and all kind of inconsistencies...

To get a clean and correct view of the mounted partitions, you should take a look to the file /proc/mounts, with something like "cat /proc/mounts". That file shows the status of the partitions as seen by the kernel, and is correct most of the times.

Note, however, that on recent kernels (2.6.x and greater), every process may have a different "view" of which file systems are mounted and unmounted. If the situation gets that much confusing, one way to understand if a partition is read-write or read-only, is just to try to "touch" a simple file... have fun :)

Manually setting the MTU allows you to force the kernel to send smaller packets regardless of the media being used or protocols like Path MTU discovery or similar.

You can set the MTU either for a whole interface, using something like 'ifconfig eth0 mtu 200' or 'ip link set eth0 mtu 200', or for a single path with something like: 'ip route add 192.168.0.0/24 via 10.0.0.1 dev eth0 mtu 200'. In this case, you need to have the 'iproute2' package installed on your system.

You may as well change the 'advertised' MSS to ask remote ends to send you smaller or bigger packets. To change the mss for a single route, just use something like: 'ip route add 192.168.0.0/24 via 10.0.0.1 dev eth0 advmss 200' if you have the iproute2 package installed, or 'route add -net 192.168.0.0 netmask 255.255.255.0 gw 10.0.0.1 mss 200'.

By using:

# netstat -s

it is possible to obtain nice-looking general purpose networking statistics, like the number of tcp connections established, failed, the number of ip packets sent, the number of packets with bad checksums, the number of segments retransmitted, and so on...

If you want to change the partition table of a given device on a running system, without rebooting, just go on and modify the partition table, paying attention not to overwrite or shrink any used partition.

Once the partition table has been changed, you need to ask the kernel to reload the partition table and update the devices accordingly. To do so, you can run the command:

# sfdisk -R /dev/sda

where sda is the name of the device where the partition table was changed. If you get the error:

BLKRRPART: Device or resource busy

it means that /dev/sda, or the device you are using, has partitions mounted on it, and the kernel was unable to read the new table. If you want to use the new partition table without rebooting, you need to umount all the old partitions of /dev/sda and then re-run 'sfdisk -R'.

If 'sfdisk -R' returns no error, it means the new partition table was read and that the devices /dev/sda1, /dev/sda2... are now relative to the new partition table you just updated.

• if you run:

  $ cat /proc/partitions

  you get the list of all partitions, as seen by the kernel. This list is usually generated at boot time and updated every time a command issues the BLKRRPART ioctl (sfdisk -R to issue the ioctl manually). This list also includes virtual devices, like those created with the LVM tools and the device-mapper.

  Note that if the partition table was changed, and the system not rebooted, or no BLKRRPART issued or issued with other partitions mounted, /proc/partitions may not reflect the content of the partition tables of the disk. Note also that if the kernel did not detect some partition table or the content of some devices, those partitions might not be available from /proc/partitions.

  Note also that any user can read the content of /proc/partitions, and that all devices in /dev representing partitions will reflect the content of this file.

• you can run:

  # fdisk -l

  or

  # sfdisk -l

  both commands will read the partition table directly from the disk. sfdisk is a very low-level tool, so, the output may slightly differ (sfdisk often shows even unused partitions).

  Note that the above commands can only be run by privileged users. Note also that the output indicates the exact content of the disk, which might not reflect the 'vision' of the kernel. If the partition table was zeroed with a 'dd' without rebooting, or the partition table was changed with a simple fdisk, with 'fdisk -l' you will see the new partition table. However, the devices in /dev, like /dev/sda1 or /dev/hdc2, will refer to the partition table as seen by the kernel, and as readable by the /proc/partitions file.

In kernel 2.4 and first 2.6, raid partitions aka md devices were not partitionable. Even today, most distributions do not support md device partitioning.

There are not many solutions to this problem, either:

• create a single raid device for the whole disk (for example, between /dev/sda and /dev/sdb), create a file system on that single partition, and live with it (mkfs.ext3 /dev/md0, where md0 is made of /dev/sda and /dev/sdb). This should have no problems, beshide that it is not very sane to put a complete system on a single partition...

• create a single raid device for the whole disk (for example, between /dev/sda and /dev/sdb), and use the device mapper (aka lvm) to create logical volumes (aka, partitions). This works, but you cannot boot from a logical volume... neither lilo nor grub support that.

• first create the partitions, and then create the raid devices. For example, you could create a small /dev/sda1 and /dev/sdb1, and a md0 device with the /boot directory on it. Then, create /dev/sda2, /dev/sdb2, /dev/sda3, /dev/sdb3 as needed and raid them separately... this solution works well, but:

  • you have to pay attention when installing the boot loader. Since only partitions are on 'raid', everything out of the partitions, like the MBR (Master boot record) is not in 'raid'. If you simply install grub or lilo as usual, at the first failure, after replacing a disk, you will likely end up with an unbootable system.

  • the solution is not very flexible. If you need to create a new partition, you have to create a new raid device, and so on... if you need to change something, it's hard to do...

  • in case of a disk failure, you will probably end up with many software raid devices not working anymore, and with the kernel having to realize that all or some of them have failed...

• Well, simply create two partitions, /dev/sda1 and /dev/sda2, raid /dev/sda1 with /dev/sdb1, and then use /dev/sda2 in raid with /dev/sdb2 as an phisica volume for LVM. That way, you can boot the system, and create new partitions simply by using the LVM.

With kernel 2.6 and a recent version of mdadm, you can also use the option --auto=mdp to create a partitionable raid volume. One more solution could be to use the EVMS, still not part of the official linux kernel.

As you can read on the Linux Software RAID HOWTO, you should set the type of raid partitions to 0xfd. Note, however, that there are two ways to assemble raid devices:

• asking the kernel to do it automatically, at boot time.

• by running mdadm or the raidhot tools right at boot time, telling them to assemble raid devices.

In practice: if you put 0xfd in the type flag of the partition table, the device will be automatically assembled at boot time. If you don't, you will need to configure mdadm and/or raidhot tools to do that for you (at boot time), or you will have to assemble the device manually.

To create the mdadm configuration file, you have two choices: one, create it manually, two, run

# mdadm --examine --scan > mdadm.conf or # mdadm --detail --scan > mdadm.conf

(the two commands give the same output, sometimes in a different order). These commands require your raid devices to be active and available. If they are not active, you can either assemble them manually, and run the above commands, or use the command:

# mdadm --examine --brief --scan --config=partitions > mdadm.conf

The standard location for the mdadm.conf file, on Debian systems, is /etc/mdadm/mdadm.conf.

Ok, let's say you have a software RAID 1 /dev/md0 device made of two partitions on two different scsi disks, /dev/sda1 and /dev/sdb1.

Let's say you just had a major hardware failure, and for one reason or another, some data was corrupted on the first device, while some other data was corrupted on the second device.

One easy way to try to recover data is to mount /dev/sda1 and /dev/sdb1 individually, recover data, and then, eventually, put them back into the raid.

Doing so, is quite easy:

• if the raid device is still mounted, umount it immediately, with something like 'umount /mount/point/of/raid/device'. You can see the mount point of the md device with something like 'mount |grep md0' or 'cat /proc/mounts |grep md0' (see note mount, read only partitions, and wierd output).

• stop the raid device, with something like:

  # mdadm --misc --stop /dev/md0

  or mark it read only, with:

  # mdadm --misc --readonly /dev/md0

• simply mount the devices independently, using:

  # mount -o ro /dev/sda1 /mnt/sda1 # mount -o ro /dev/sda2 /mnt/sda2

  making sure the /mnt/sda1 and /mnt/sda2 directory exist.

• You can now work on /mnt/sda1 or /mnt/sda2 without problems (...).

• if you later decide that the content of one of the partitions is good, you can start the array in degraded mode with --assemble, or mark /dev/sda2 (for example) as failed, and then simply replace it ...

Ok, so... you are switching from a non-LVM system to using LVM... you have your /dev/sdb, and want to turn it in a Physical Volume to add to a simple Volume Group.

You try with a simple:

# pvcreate /dev/sdb

and you get an error:

Can't open /dev/sdb exclusively. Mounted filesystem?

you check the mount option, with something like:

mount |grep /dev/sdb

and nothing appears... (well, if something appears, just run umount all mounted partitions, and try one more time). So: pvcreate fails, reporting 'Mounted filesystems?', but no file systems are mounted.

The solution is quite simple: as long as the kernel sees partitions on the device, pvcreate will not be able to lock it to create a physical volume... (this is still true on a 2.6.16 with lvm2 2.02.05). So, remove the partitions (with cfdisk/sfdisk/fdisk, whatever you want, or dd if=/dev/zero of=/dev/sdb size=512 count=1) run 'sfdisk -R' and you should be fine... run pvcreate, and you will have no more errors...

# dd if=/dev/zero of=/dev/sdb size=512 count=1 # sfdisk -R # pvcreate /dev/sdb

To backup a partition table, you can simply run:

# sfdisk -d /dev/sda > /home/backup.file

To restore the backup, which means, to repartition the disk as the dump you just generated, you can simply run something like:

# sfdisk /dev/sda < /home/backup.file

Note that the above commands do not save/restore the full MBR: if a boot loader or something similar is installed, its own code will be lost... you may have better luck by using dd, but watchout with extended partitions...

To backup the partition table using dd, simply run:

# dd if=/dev/sda of=/tmp/backup.pt bs=512 count=1

where /dev/sda is the device of which you want to backup the partition table, /tmp/backup.pt is the name of the file where you want your partition to be stored, and bs=512, count=1, tells dd to copy 1 sector of 512 bytes from the beginning of the disk...

This method will backup the full MBR, both the partition table and a small fragment of code used to boot your system in front of the partition table. However, IT WILL NOT BACKUP EXTENDED PARTITIONS!

This means that if you have /dev/sda5, /dev/sda6 or greater... the partition table records about those partitions will be lost. A better solution would be to save both the first sector of the disk using dd, and then the complete backup of the partition table using 'sfdisk -d' (take a look at http://notes.inscatolati.net/system[en]/storage[en]/index.html#29).

So, you have just bought 10/15 bleeding edge scsi disks, of the same type, brand, and of the same kind, and you want to create a partition table on each of them?

A simple way would just be to create a partition table on the first device (sda), with cfdisk, fdisk or whatever you like, and then backup the partition using:

# sfdisk -d /dev/sda > /tmp/backup.pt

Now, for each device, you can simply run:

# sfdisk /dev/sdb < /tmp/backup.pt

In short, something like:

# cfdisk /dev/sda # sfdisk -d /tmp/backup.pt # for dev in /dev/sd{b,c,d,e,f,g,h,i,j,k}; do \ > sfdisk $dev < /tmp/backup.pt; done

should be enough...

At least twice in my life I've been surprised by grep either not matching what it was supposed to, or matching too much.

Both times it turned out to be a problem with the locale configured on the system. Yes, locale settings influence things like what are to be considered whitespaces, letters, and so on. grep and many other matching libraries keep that in consideration, and change behavior accordingly.

While this is kind of expected, if the locale configuration is screwed for the language being used, grep (and many other tools...) may really get confused.

Check the locale configurations, and try with things like:

LC_ALL=C grep ...

to see if the problem goes away or grep changes behavior. If it does change, well... you know for sure the problem is with the locales. Good luck with the debugging :)

The whole message looks something like:

Setting up LVM Volume Groups... Reading all physical volumes. This may take a while...

followed by lot of kernel messages.

Well, that's the problem: lvscan, to find all the phyisical volumes, will just scan the first few sectors of every device on the system.

If you happen to have something particularly slow, lot of devices, or ... you are an aficionado of devfs (which will probe modules as pvscan tries to find those volumes), you will want to change /etc/lvm.conf.

Just add something like:

filter = [ "a|/dev/md*|", "a|/dev/sd*|" ]

in that file, to tell pvscan to search only in /dev/md* and /dev/sd*

Is fairly easy, just run something like:

/usr/sbin/dovecot --exec-mail pop3 2>/dev/null

if you have enough privileges to read the conf file, you'll be done.

This is useful, for example, if you have a system storing mail in /var/spool/mail, ~/Maildir/ or ~/mbox, and you want to download it with fetchmail, but no official pop3/imap is configured. Get a dovecot binary, a conf file, and configure fetchmail with something like:

plugin "ssh server -- echo +OK; exec /usr/sbin/dovecot --exec-mail pop3 2>/dev/null" auth ssh

Yes, this is a hack, and yes, there are plenty of other ways to achieve the same result. But it works, guarantees an encrypted channel, uses ssh authentication, and doesn't require you to configure dovecot (or any imap / pop3 server) to do authentication for real.

By default, the linux kernel will output lot of different kind of messages on your console. To disable them, you can run:

dmesg -n1

to tell the kernel to only send panic messages to your console. This is useful, for example, when you have enabled log martian in the kernel, added some -j LOG in iptables, or you have some hardware problem that's causing the kernel to spam you.

If you don't have a prompt, or it's too hard to write, and have emergency sysctl enabled in your kernel, you can try pressing ctrl+alt+stamp+1 (aka ctrl+alt+sysreq+1).

If you want this parameter to be automatically set at every reboot, you can add the -c1 parameter to klogd, in /etc/init.d. To do so in Debian, you need to modify /etc/default/klogd, to have something like:

KLOGD="-c1 [whatever parameter was already there]"

Note that even though this parameter is set to 1, the messages will still go to syslogd, which can decide that the message is important enough to be outputted in your console. Either stop syslogd temporarily, or change /etc/syslog.conf if it bothers you.

Ok, let's say you have created your ext3 root filesystem, and you want to use it with data=journal, instead of the default data=ordered. You cannot umount it, so you try:

mount -o remount,data=journal

Most likely, the command will fail with:

mount: / not mounted already, or bad option

If you also run dmesg, you will see the error:

EXT3-fs: cannot change data mode on remount

so you're stuck. Also, the root partitions gets mounted before fstab is read, so... it's a bit harder to get the options right.

If you cannot reboot the system, the only thing you can try is to play with pivot_root or similar, to try to temporarily move the root on tmpfs / ramdisk / ... but would still need lot of restarting, mount --bind and so on to move the services to the new root.

If you can afford to reboot, and you want data=journal after every reboot, you can use:

tune2fs -o journal_data /dev/sda1

where sda1 is your root partition. Alternatively, on LILO or grub command line, you can pass the option:

rootflags=data=journal

Just:

• check how many extents you have available in the volume group

• use lvcreate with the -l option to specify the number of extents

   % vgs -o vg_name,vg_size,vg_free,vg_extent_count,vg_free_count
   VG     VSize  VFree #Ext  Free
   system 67.96G 8.96G 17397 2293
   % lvcreate -n backup -l 2293 system
   

Instead of using vgs, you can use vgdisplay, and use the field:

[...] Free PE / Size 2293 / 8.96 GB [...]

Just:

• check how many extents you have available in the volume group

• use lvcreate with the -l option to specify the number of extents

   % vgs -o vg_name,vg_size,vg_free,vg_extent_count,vg_free_count
   VG     VSize  VFree #Ext  Free
   system 67.96G 8.96G 17397 2293
   % lvcreate -n backup -l 2293 system
   

Instead of using vgs, you can use vgdisplay, and use the field:

[...] Free PE / Size 2293 / 8.96 GB [...]

mkinitramfs allows you to specify some parameters on the LILO or GRUB prompt to easily (sure) debug problems. Most useful parameters are probably:

• debug, to have all the shell scripts on the initrd run with the -x parameter, and the output logged in /tmp/initramfs.debug on the ramdisk. To have the output sent to stdout, specify something like debug=/dev/console.

• break, to have the initrd return a shell prompt. If no parameters are specified, the prompt will be returned when most convenient to the initrd (at current time, just before mounting the root filesystem - premount). Otherwise, you can specify something like break=whatever, to ask the initrd to stop at exactly the specified step. Steps currently defined by mkinitramfs are: top, modules, premount, mount, bottom and init.

• blacklist, if you suspect a module is causing problems to your hardware. Specify something like blacklist="module1 module2" to have module1 and module2 not loaded at boot time.

As I use grub on my system, to specify those parameter at boot time, I usually press 'e' on the line I usually boot from. Once there, I modify the line:

kernel /vmlinuz root=/dev/mapper/root ro

to have the options I need:

kernel /vmlinuz root=/dev/mapper/root ro break=top

and finally press 'b' to get the system booted with the specified parameters.